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Abstract 


The Rural Math Innovation Network (RMIN) is a 4-year project that launched in January 
2017 after receiving a $2.9 million Investing in Innovation (i3) development grant from the U.S. 
Department of Education (ED) and matching funds from the private sector. Virginia Ed Strategies 
and rural local education agencies (LEAs) in Virginia are implementing a project using a networked 
improvement community (NIC) of Pre-Algebra and Algebra 1 teachers to incorporate social- 
emotional learning (SEL) factors of academic self-efficacy and growth mindset into lesson plans for 
teaching career readiness math competencies. 


During Year 1, the project established Memos of Understanding with 18 school divisions in 
southwest and southside Virginia, which enabled math teachers within these divisions to submit 
applications to participate in the project. At the end of Year 1, December 2017, the project had a 38- 
member teacher cohort across 25 schools. By the end of Year 2 (January 1 - December 31, 2018), 
the cohort included 30 teachers (19 middle school teachers and 11 high school teachers) across 20 
schools (12 middle schools and 8 high schools) within 16 participating divisions (several teachers 
dropped out of the project in Year 2 and a few teachers were added). During Year 3 (January 1 - 
December 31, 2019), several more teachers dropped out of the project, resulting in 26 teachers (17 
middle school and 9 high school) across 18 schools within 15 participating divisions. Sixteen of the 
26 teachers are located in the southside region, with the remaining 10 teachers located in the 
southwest region of Virginia. 


One of the i3 requirements is to have an external evaluation conducted of the project; 
development grants must include both an implementation study and an impact study. To fulfill this 
requirement, Virginia Ed Strategies hired ICF to conduct an independent evaluation of the RMIN 
project throughout the 4-year period. The evaluation includes three components: a formative study 
to provide ongoing feedback about participants’ reactions, learning, behaviors, and results; an 
implementation study focusing on how well the structural and programmatic aspects of the RMIN 
project are implemented, as well as facilitating or impeding factors; and an impact study to 
determine the extent to which the project impacts high-need students’ math achievement. 
Previously in Year 2, the evaluation team recruited 10 comparison teachers across rural Virginia 
school divisions who were also teaching either Pre-Algebra or Algebra 1 (no school has both a 
participating and comparison teacher). 


The purpose of this report is to summarize key findings from the impact study. The primary 
audience is the RMIN project staff at Virginia Ed Strategies; secondary audiences include ED and 
other interested stakeholders. Although the impact study was designed originally to include 
students from the 2018-19 and 2019-20 school years, the COVID-19 pandemic in spring 2020 led to 
school closings and no administration of the statewide Standards of Learning (SOL) math 
assessment. Therefore, the impact study is based solely on the one year of SOL data. 


Findings are presented for the impact data and framed by the evaluation questions. 
Conclusions are presented below. 


Impact on student achievement. The one-year program impact on students’ SOL scores 
was estimated and the results did not find evidence that the RMIN program significantly improved 
students’ SOL performance. The program impact from the Pre-Algebra sample was negative, but it 
was not statistically significant, and the effect size was small. The program impact from the Algebra I 
analysis was positive but not statistically significant, and the effect size was small. 


Introduction 


Established in 2009 under the American Recovery and Reinvestment Act, the Investing in 
Innovation (i3) fund promotes public and private investments in local education agencies (LEAs) 
and non-profit organizations to improve student achievement and attainment in low-income 
communities, and create an education sector supportive of the rapid development and adoption of 
effective solutions.1 These competitive grants are awarded to school districts, nonprofit 
organizations working with districts, or a consortium of schools with a record of improving student 
achievement and attainment, and demonstrated public-private commitments. 


RMIN Project Overview 


The Rural Math Innovation Network (RMIN) is a 4-year project that launched in January 
2017 after Virginia Ed Strategies received a $2.9 million i3 development grant from the U.S. 
Department of Education (ED) and matching funds from the private sector.2 Virginia Ed Strategies 
and rural LEAs in Virginia are implementing a project using a networked improvement community 
(NIC) of Pre-Algebra and Algebra I teachers to incorporate social-emotional learning (SEL) factors 
of academic self-efficacy and growth mindset into lesson plans for teaching career-readiness math 
competencies. The RMIN objectives are: 


1. To prepare all teachers in the NIC to innovate lesson plans with SEL strategies that 
address self-efficacy and growth mindset needs of students for learning mathematics 
required in STEM-H technician careers; 


2. To provide supports for each teacher in the NIC to innovate 5 SEL math lessons and 
implement 10 SEL math lessons into instruction; 


3. To establish technology capacity for NIC teachers to develop and implement SEL math 
lessons; 


4. To achieve a student passage rate increase of 25% on the career-readiness math 
assessments: Virginia Pre-Algebra test, Virginia Algebra I test, Work Keys math test, and 
Virginia Community College math assessment; and 


5. To broadly disseminate information that results in four schools as teacher innovation 
sites and a 125% increase in the NIC math teacher membership. 


During Year 1, the project established Memos of Understanding with 18 school divisions in 
southwest and southside Virginia, which enabled math teachers within these divisions to submit 
applications to participate in the project. At the end of Year 1, December 2017, the project had a 38- 
member teacher cohort across 25 schools. By the end of Year 2 (January 1 - December 31, 2018), 
the cohort included 30 teachers (19 middle school teachers and 11 high school teachers) across 20 
schools (12 middle schools and 8 high schools) within 16 participating divisions (several teachers 
dropped out of the project in Year 2 and a few teachers were added). During Year 3 (January 1 - 
December 31, 2019), several more teachers dropped out of the project, resulting in 26 teachers (17 


1 See https:/Avww2.ed.gov/programs/innovation/index.html 
2 See http://docs.wixstatic.com/ugd/19fc5e_55a5434194304978bcc6f1f88e4a80d8.pdf 


middle school and 9 high school) across 18 schools within 15 participating divisions. Sixteen of the 
26 teachers are located in the southside region, with the remaining 10 teachers located in the 
southwest region of Virginia. 


RMIN Evaluation 


One of the i3 requirements is to have an external evaluation of the project; development 
grants must include both an implementation study and an impact study. To fulfill this requirement, 
Virginia Ed Strategies hired ICF to conduct an independent evaluation of the RMIN project 
throughout the 4-year period. The evaluation includes three components: a formative study, an 
implementation study, and an impact study. 


The formative study provides project staff with ongoing feedback as the teachers 
participate in the RMIN project. These findings can be used to make programmatic adjustments as 
needed to better meet participants’ needs. The formative focus is on participants’ reactions, 
learning, behaviors, and results. This study will be carried out all four years. 


The implementation study investigates how well the structural and programmatic aspects 
of the RMIN project are being implemented, i.e., the fidelity of implementation. Fidelity focuses on 
adherence, dosage, quality of delivery, and participant responsiveness. This study was carried out 
in the first two years of the RMIN project. 


The impact study determines the extent to which the project impacts high-need students’ 
math achievement. This study employs a quasi-experimental design with comparison teachers so 
that the math achievement of students within treatment or comparison classrooms can be 
examined. This study began in Year 2, with the securement of comparison teachers so that student 
data could begin being collected. Although the impact study was designed originally to include 
students from the 2018-19 and 2019-20 school years, the COVID-19 pandemic in spring 2020 led to 
school closings and no administration of the Virginia statewide Standards of Learning (SOL) math 
assessment. Therefore, the impact study is based solely on the one year of SOL data. 


The purpose of this report is to summarize key findings from the impact study. The primary 
audience is the RMIN project staff at Virginia Ed Strategies; secondary audiences include ED and 
other interested stakeholders. 


Evaluation Design and Methodology 


Evaluation Design 


As noted earlier, ICF is conducting an external evaluation of the RMIN project that includes 
a formative study to provide ongoing feedback, an implementation study to investigate fidelity, and 
an impact study to determine the project impact on students’ math achievement outcomes. The 
impact study design is described below. 


This section focuses on the quasi-experimental study evaluating the RMIN program’s impact 
on student mathematics achievement. As of January 2020, the ICF team has collected data to 
address evaluation questions for first-year students taught by participating and comparison 
teachers in Pre-Algebra and Algebra I courses. 


This analysis focused on two of the four impact evaluation questions.? 


1. What is the effect of the RMIN intervention on the math achievement of Pre-Algebra 
takers compared to the math achievement of Pre-Algebra takers in the business-as-usual 
condition at the end of Year 2? 


2. What is the effect of the RMIN intervention on the math achievement of Algebra | takers 
compared to the math achievement of Algebra I takers in the business-as-usual condition 
at the end of Year 2? 


Year 2 indicates that this was the second year of intervention for treatment (participating) teachers, 
which included student pretest (baseline) data from the 2017-18 school year and posttest data 
from the 2018-19 school year. This was the first cohort of students who were in the Pre-Algebra or 
Algebra I courses taught by treatment or comparison teachers during school year 2018-19. 


Data and variables. The data from the first cohort of students were provided to the ICF 
team by the Virginia Department of Education (VDOE) at the end of December 2019. To address the 
evaluation questions, the analysis required the SOL test scores from one year prior to the 
intervention year (school year 2017-18) and the scores from the intervention year (school year 
2018-19). To adjust for the influence of other student characteristics in the program impact 
analysis, demographic data were also obtained from the VDOE. The student variables provided 
were grade level, minority status (white or minority), English language learner status, and 
economic disadvantage status (based on free/reduced-price meal eligibility, TANF‘ recipient status, 
and Medicaid eligibility). Gender was not requested to address data security and privacy concerns. 


Table 1 summarizes the number of cases per each raw dataset. The pretest data and the 
posttest data included, respectively, 2,784 cases and 2,598 cases with no missing values on analysis 
variables. After combining these datasets and keeping only Pre-algebra or Algebra I students in the 
sample, the data sample consisted of 2,255 students. When only Pre-Algebra courses were selected, 
the number of valid cases was 1,063. When Algebra I courses were selected, the number of cases 
was 1,192. As discussed below, these numbers were further reduced to address the data challenge 
of students taking different types of mathematics tests and students’ grade levels. 


Table 1. Number of Students in Raw SOL Databases 


Databases Number of Students 
School year 2017-18 SOL pretest data (administered Spring 2018) 2,784 
School year 2018-19 SOL posttest data (administered Spring 2019) 2,598 


Combined data - cases with no missing values on any of the 
variables used in the analysis (keeping only students who were 2,255 
enrolled in either Pre-Algebra or Algebra I courses) 


Number of students in Pre-Algebra courses in the combined dataset 1,063 


Number of students in Algebra I courses in the combined dataset 1,192 


3 These were initially intended as the exploratory questions, with the third and fourth evaluation questions 
serving as the confirmatory questions, focusing on student data from project Year 3. Given the COVID-19 
pandemic, SOL data for school year 2019-20 were unavailable. 

4 Temporary Assistance for Needy Families (TANF) 


In constructing the analysis sample from the raw databases (Pre-Algebra dataset, n=1,063; 
Algebra | dataset, n=1,192), there were some challenges and limitations. The first consideration 
was that students took tests in different mathematical areas (Mathematics 7, 8, Algebra I, and 
Algebra II). The decision was made to focus on Pre-Algebra students who took Mathematics 7 test 
as pretest and Mathematics 8 test as posttest and on Algebra I students who took Mathematics 8 
test as pretest and Algebra I test as posttest. The chosen posttest test types are consistent with the 
research interest (Pre-Algebra and Algebra I instruction and student achievement) and the chosen 
pretest maximized the resulting sample sizes and optimized statistical power. The other 
consideration was the grade levels of students. The decision was made to focus on only eighth 
graders for the Pre-Algebra analysis and only ninth graders for the Algebra I analysis, which again 
maximized possible sample sizes and provided more balance between the treatment and 
comparison group sample sizes. See Appendix A for more discussion of these data decisions. 


As a result of these decisions, the number of teachers and students included in these final 
datasets were smaller than the original raw datasets. The Pre-Algebra sample consisted of students 
taught by ten treatment teachers from eight schools and two comparison teachers from two 
schools. The Algebra I analysis sample consisted of students taught by nine treatment teachers from 
six schools and six comparison teachers from six schools. Table 2 describes the final analytic 
samples. 


Table 2. Final Analytic Samples 
PRE-ALGEBRA SAMPLE (eighth graders only) 


Treatment Group Comparison Group Combined 
Schools 8 Schools 2 Schools 10 
Teachers 10 Teachers 2 Teachers 12 
Students 432 Students 80 Students 512 


ALGEBRA I SAMPLE (ninth graders only) 


Treatment Group Comparison Group Combined 
Schools 6 Schools 6 Schools 12 
Teachers 9 Teachers 6 Teachers 15 
Students 327 Students 193 Students 520 


Note: Pre-Algebra students in the sample (all eighth graders) took Mathematics 7 and Mathematics 8 SOL tests, 
respectively, for pretest and posttest. Algebra students in the sample (all ninth graders) took Mathematics 8 SOL test and 
Algebra I SOL test, respectively, for pretest and posttest. 


The second data challenge was the fact that there were fewer comparison teachers than 
treatment teachers, resulting in less student data for comparison classrooms. This particularly 
imposed a constraint for the Pre-Algebra analysis sample as not many comparison teachers taught 
Pre-Algebra courses. As explained in Appendix A, this became a factor in how the two analysis 
samples were defined. 


Baseline equivalence of analysis samples. To evaluate the program effectiveness, 
comparison teachers were recruited from the same or similar school districts and the ICF team 
employed the Propensity Score Matching (PSM) analysis. The PSM analysis compared treatment 


students and comparison students based on prior-to-the intervention characteristics, keeping only 
matched pairs of treatment and comparison students in the analysis sample. This study used 
pretest SOL scores as the baseline variable used in the PSM analysis. The scores used for this 
analysis were z-scores (standardized with a sample mean of 0 and standard deviation of 1). 


Even without PSM, as shown in Table 3, the analysis samples established baseline 
equivalence in pretest SOL scores. The standardized pretest group difference was -.21 and 0.19, 
respectively, for the Pre-Algebra sample and the Algebra I sample. Per What Works Clearinghouse 
(WWC) guidelines, these values were within the acceptable range. WWC requires that the covariate 
whose group differences were greater than 0.05 and smaller than 0.25 be included in the final 
statistical model. Pretest SOL scores were used as a covariate in the statistical model. Because the 
samples already pass the WWC guideline for baseline equivalence, the ICF team decided to use the 
original sample without relying on the PSM analysis. 


Table 3. Baseline Equivalence Analysis of Pretest SOL Test Scores 


N Raw Raw Standard- | Standard- orreiaaaaa WWC Test 
Mean SD ized Mean ized SD ; Result 
(Hedge’s g) 

Pre-Algebra (all eighth graders; n=512) 
Treatment 
eee 432 389 51 -0.03 1.01 Baseline 

: -0.21 equivalence 
COMPANeOR | 5) a0 48 0.18 0.95 established 
students 
Algebra I (all ninth graders; n=520) 
Treatment 
cee 327 423 39 0.07 0.98 Baseline 

; 0.19 equivalence 
Comparison.) a3) 445 42 0.19 1.03 established 
students 


Note: The absolute values of estimated standardized effects were greater than 0.05 but smaller than 0.25; the two analysis 
samples establish baseline equivalence with a requirement that pretest scores must be included in the statistical model 
(per WWC guideline). 


Statistical analysis. The goal of the hierarchical linear model (HLM) analysis is to estimate 
the impact of RMIN intervention on student SOL test scores. The statistical model used posttest SOL 
test scores as the outcome and controls for pretest SOL test scores, as well as other student 
predictors: racial majority-minority status (white vs. non-whites), English language learner status, 
and economic disadvantage status (based on free/reduced-price meal eligibility, TANF recipient 
status, and Medicaid eligibility). Grade levels were not used because the Pre-Algebra sample 
included only eighth graders and the Algebra I sample included only ninth graders. The program 
effect was estimated as the coefficient of the group status (1 if treatment, 0 if comparison) and the 
standardized effect size was presented to facilitate interpretation. The standardized program effect 
was derived by running the statistical model using the z-score version of posttest SOL scores (z- 
score used the sample mean and sample standard deviation (SD); not the state mean or state SD). 


To address the clustering issue inherent in education data (students are nested within 
clusters and thus errors are not independently distributed), the model estimated the intercepts (i.e., 


teacher effects) as random effects. The rational for treating teachers as the nesting unit comes from 
the fact that RMIN recruited individual teachers for program participation and the study was 
designed as a teacher-level study. Preparatory analysis suggested that using the school level as the 
nesting unit in the model does not substantially change the analysis result. 


The following equation summarizes the model described above. 


Posttest,, = Bw + Bw* pretest,, 
+ B»* treatment , Pht Pu; 


where 

= Posttest represents posttest SOL scores 

= Pretest represents the pretest SOL scores 

"  Postscripts i and j, respectively represent student and teacher 
= fs are parameters to be estimated andr and u are error terms 


= The three ellipses (i-e., “...”) indicate that the model will include multiple predictors (white 
student, English language learner, and economic disadvantage status) and corresponding 
parameters 


« Treatment represents the group status (1 if treatment group; 0 if comparison group) 


Findings 


The following section provides findings for the impact study, framed by the evaluation 
questions. This summary is based on data from student math achievement scores. 


Impact Evaluation Findings 


The following section provides a summary of findings for the exploratory questions for the 
impact study. Data sources include statewide math assessment scores (Standards of Learning, or 
SOL) for students in treatment and comparison teacher Pre-Algebra or Algebra I classrooms during 
the 2018-19 school year. 


Tables 4 and 5 summarize the result of the program impact analysis, respectively, for Pre- 
Algebra and Algebra | students. As mentioned in the method section, HLM was used to analyze the 
data. The treatment coefficient, which corresponds to the program impact, was adjusted for race 
(minority vs. white), students’ English language learner status, and economic disadvantage status 
(defined earlier). Descriptive statistics for the samples used for analysis can be found in Appendix B 
(Table B1 and B2 were, respectively, for the Pre-Algebra analysis and the Algebra | analysis). 


The results from the Pre-Algebra and Algebra I analyses both show that the program impact 
was small. The standardized effect for the Pre-Algebra analysis was -0.17 and the effect for the 
Algebra I analysis was 0.07. The negative effect for the Pre-Algebra, however, was not statistically 
significant and the effect size was relatively small. WWC considers an effect size greater than 0.25 
as “substantively important.” The positive effect from the Algebra I analysis is consistent with the 
program expectation. The effect size, however, was also small and not statistically significant. 


Table 4. Pre-Algebra: Summary of Program Impact HLM Analysis: Adjusted Posttest SOL Test 
Average Scores by Group Status (n=512) 


Standard Standardized 
Estimate Error P-value Stat. test Program 
Impact 
Intercept 436.65 8.99 <.0001 nae 
Treatment -7.90 9.20 0.41 ns -0.17 
Pretest (z-score) 35.16 1.39 <.0001 nee 
White student -2.00 3.19 0.53 ns 
Brgrsi Ianeueee 13.05 11.11 0.24 ns 
learner 
paces -12.96 3.19 <.0001 so 
disadvantage 


Note: Significance test: ns if p >=10 ~ if p<0.10, * if p< 0.05; ** if p< 0.01, *** p< 0.001. See Appendix B, Table B1 for 
descriptive statistics of the sample used for this analysis. See Table B3 for unadjusted descriptive statistics of posttest 
scores. 


Table 5. Algebra I: Summary of Program Impact HLM Analysis: Adjusted Posttest SOL Test 
Average Scores by Group Status (n=520) 


Standard Standardized 
Estimate P-value Stat. test Program 
Error 
Impact 
Intercept 430.90 9.26 <.0001 mee 
Treatment 2.40 11.61 0.84 ns 0.07 
Pretest (z-score) 18.02 1.35 <.0001 mee 
White student -4.45 2.93 0.13 ns 
English language 8.36 8.19 0.31 ns 
learner 
Economie 0.44 3.49 0.90 ns 
disadvantage 


Note: Significance test: ns if p >=10 ~ if p<0.10, * if p< 0.05; ** if p< 0.01, *** p< 0.001. See Appendix B, Table B2 for 
descriptive statistics of the sample used for this analysis. See Table B3 for unadjusted descriptive statistics of posttest 
scores. 


Figure 1 expresses the same information graphically. To reiterate the findings, the Pre- 
Algebra analysis shows the comparison group performed better than the treatment students; 
however, the difference between the treatment and the comparison group was small (standardized 
effect of -0.17) and not statistically significant. For the Algebra I analysis, the treatment group 
performed slightly better than the comparison group; however, again the group difference was 
small and not statistically significant (standardized effect of 0.07). 


450 
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Pre-Algebra: Pre-Algebra: Algebra |: Algebra |: 
Treatment group Comparison group Treatment group Comparison group 
(n=432) (n=80) (n=327) (n=193) 


Note: The average score for the comparison group was fixed at the unadjusted means of the comparison group 
(Pre-Algebra 432.97; Algebra I 444.98). 


Figure 1. Average Posttest SOL scores (Adjusted for Covariates) 


Conclusions 


Conclusions pertaining to the impact study are presented below. These conclusions are 
derived from analysis of available data sources. 


Impact on Student Achievement 


The one-year program impact on students’ SOL scores was estimated and the results did not 
find evidence that the RMIN program significantly improved students’ SOL performance. The 
program impact from the Pre-Algebra sample was negative, but it was not statistically significant 
and the effect size was small (standardized effect size, -0.17). The program impact from the Algebra 
I analysis was positive but not statistically significant, and the effect size was small (standardized 
effect size, 0.07). The result involving the first cohort of students does not negate the expectation 
that RMIN improved student performance. The samples included a small number of comparison 
teachers’ students and thus this might have posed an estimation challenge. The fact that students 
often took different types of SOL tests and that the analysis focused on the exact pretest-posttest 
type combination further reduced the number of cases. 


Appendix A: Data Definition of the Program Impact SOL Analysis Samples 


This section describes how the program impact analysis samples were defined. Table A1 
summarizes what SOL test students in the sample took for pretest and posttest. The largest 
number of students (n=513) took Mathematics 7 as pretest and Mathematics 8 as posttest. Other 
combinations of pretest and posttest shown in other columns were not only smaller in number of 
students, but data were more unbalanced by group status. For example, the number of students 
who took Mathematics 6 as pretest and Mathematics 7 as posttest was 268; however, all 268 
students were treatment students. Other columns exhibit the same problem of how all or almost all 
students were treatment students. Including these unbalanced datasets would likely bias the 
program impact. Therefore, the Pre-Algebra sample included only those who took Mathematics 7 as 
pretest and Mathematics 8 as posttest. 


Table A1. The Type of SOL Test Pre-Algebra Students Took for Pretest and Posttest by Group 
Status 


Pretest SOL Posttest SOL Treatment Comparison Total Notes 
Test Type Test Type Group Group 
Math 6 Math 7 268 0 268 
Math 6 Math 8 274 0 274 
Math 7 Algebra | 1 0 1 
Math 7 Math 8 433 80 513 mee 
sample 
Math 8 Math 8 6 1 7 
Total 982 81 1,063 


We took a closer look at the sample consisting of those who took Mathematics 7 for pretest and 
Mathematics 8 for posttest. As shown in Table A2, almost all students (except for one) in the sample 
were eighth graders. The analysis sample thus excluded the single seventh grader and included 
only eighth graders. The final sample size, therefore, was 512. 


Table A2. Grade Levels of Students who Took Math 7 and Math 8 SOL Tests 


Group Status Seventh Graders Eighth Graders 
Excluded from the Sample Included in the Sample 
Comparison Group 0 80 
Treatment Group 1 432 
Total 1 512 


Table A3 shows what Algebra | students took for pretest and posttest and the results were again 
separated by group status. The largest number of students took Mathematics 8 as pretest and 
Algebra I as posttest. Another combination, Mathematics 7 as pretest and Algebra I as posttest, had 
a relatively larger number of students than the rest of the combinations which barely had any cases. 


The decision was made to focus only on the Mathematics 8 and Algebra I sample to make 
interpretation clear and to parallel the Pre-Algebra decision noted above.5 


Table A3. The Type of SOL Test Algebra I Students Took for Pretest and Posttest by Group 
Status 


Pretest SOL Posttest SOL Treatment Comparison Total Notes 
Test Type Test Type Group Group 
Algebra | Algebra | 16 16 32 
Geometry Algebra | 5 1 6 
Geometry Algebra II 1 0 
Geometry Geometry 0 1 1 
Math 6 Algebra | 2 0 2 
Math 7 Algebra | 212 86 298 
Math 7 Math 8 7 0 7 
Math 8 Algebra I 635 210 845 ae 
Total 878 314 1,192 


The decision of including only ninth graders stemmed from the following. Table A4 shows how the 
number of Algebra I students vary by group status and grade levels. The ninth grade subsample 
includes relatively large number of students both in the treatment and comparison groups. The 
data are unbalanced for seventh graders (0 comparison students; 12 treatment students) and 
eighth graders (17 comparison students and 296 treatment students). This could bias the result of 
program impact estimation and thus we decided to focus on ninth graders for the Algebra analysis. 


Table A4. Grade Levels of Students Who Took Math 8 and Algebra I SOL Tests 


Seventh Graders Eighth Graders Ninth Graders 
Excluded from the Sample | Excluded from the Sample | Included in the Sample 


Comparison Group 0 17 193 
Treatment Group 12 296 327 
Total 12 313 520 


5 An alternative approach was to combine the two groups and standardize the pretest test scores using 
the Virginia state average score and standard deviation reported per each test type; however, this relies 
on the assumption that the Math 7 SOL test and the Math 8 SOL test were comparable if standardized 
(meaning, for example, students with the average Math 7 score and students with the average Math 8 
test score have the same level of math competency). The decision was made to not employ this 
alternative approach since state statistics were not available at the time of this report. 
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Appendix B: Descriptive Statistics Tables for the Program Impact SOL Analysis 
Table B1. Descriptive Statistics for the Sample Used for Pre-Algebra HLM Analysis (eighth graders) 


N Mean SD Min Max 
Treatment 512 0.84 0.36 0.00 1.00 
SOL Math 8 score (Posttest) 512 426.33 46.15 223.00 600.00 
SOL Math 8, z-score 512 0.00 1.00 -4.41 3.76 
SOL Math 7 score (Pretest) 512 391.01 50.88 206.00 600.00 
SOL Math 7, z-score 512 0.00 1.00 -3.64 4.11 
White student 512 0.72 0.45 0.00 1.00 
English language learner 512 0.01 0.11 0.00 1.00 
Economic disadvantage 512 0.19 0.40 0.00 1.00 


Table B2. Descriptive Statistics for the Sample Used for Algebra I HLM Analysis (ninth graders) 


N Mean SD Min Max 
Treatment 520 0.63 0.48 0.00 1.00 
SOL Algebra I score (Posttest) 520 433.25 35.29 337.00 569.00 
SOL Algebra I, z-score 520 0.00 1.00 -2.73 3.85 
SOL Math 8 score (Pretest) 520 420.04 40.42 288.00 538.00 
SOL Math 8, z-score 520 0.00 1.00 -3.27 Z.92 
White student 520 0.67 0.47 0.00 1.00 
English language learner 520 0.02 0.15 0.00 1.00 
Economic disadvantage 520 0.16 0.37 0.00 1.00 
Table B3. Descriptive Statistics of Posttest SOL Test Scores Used for the Final HLM Analyses 
Raw Raw | Standard- | Standard- Staucaneuzed Statistical 
N : ; Difference meets 
Mean SD ized Mean ized SD Fi Significance 
(Hedge’s g) 
Pre-Algebra (all eighth graders; n=512) 
Treatment 
dea 432 423 44 -0.07 0.95 Statistically 
- -0.48 significant at 
Compensoh:| ao! ate |||, 6a 0.40 1.46 p=.001. 
students 
Algebra I (all ninth graders; n=520) 
preaunent. 397 | 493 36 0.00 1.03 
students Not 
: ROO significant 
Compan S08) 99:|° 3434 34 -0.00 0.95 
students 


Note: The results are based on simple descriptive statistics. While the Pre-Algebra result is statistically significant, the 
multilevel multivariate statistical model found the adjusted average difference to be not statistically significant (see Table 4). 
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